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INTRODUCTION 



Interest in the potential use of satellites for communications has grown steadily the past 
few years. Over 40 satellite-related projects, in various stages of planning and development, 
currently are progressing in countries, such as Indonesia, Brazil, Iran, and India, as well as 
in the United States. 

Many of these projects, however, are proceeding without the benefit of accurate informa- 
tion about the capabilities of satellite-based media systems or about the problems entailed in 
operating these systems. There is a pressing need for empirical data. 

It was precisely to address this need that the Satellite Technology Demonstration (STD) of 
the Federation of Rocky Mountain Stages, Incorporated was Instituted. From 1972-75, the STD 
combined technology, courseware, and human support elements to provide a variety of educational 
services to rural, isolated populations in the Rocky Mountain Area. Project sponsors believed 
that this prototypic effort would furnish essential information on the development and imple- 
mentation of satellite communications systems for future decision-makers planning similar 
projects. 

Now that the STD has completed its operational period, documentation products have been 
developed to distribute the findings. This report Is part of that documentation. Instead of 
focusing on the findings, however, the report describes the process used by the STD to design 
and conduct research efforts. 

THE PROBLEM 

A telecommunication satellite, such as the ATS-6 used by the STD, is extremely versatile, 
complex, and expensive; its potential is enormous. The research effort surrounding this sat- 
ellite is, in turn, large and complex. The STD's Project, for example, Involved many agen- 
cies: the National Aeronautics and Space Administration; the Department of Health, Education, 
and Welfare; the Federal Communications Commission; the Interagency Radio Advisory Committee; 
the National Institute of Education; the departments of education and their respective bureaus 
in eight participating states; public television stations In the Rocky Mountain Area; and 
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loca^ school districts, instructional staff, and students. This mix of direct participants and 
involved agencies helped to shape the STD's Project— its goals and its research effort, both of 
which are interrelated. 

The STD used part-time consultants to obtain research input during its early planning and 
proposal activities. No central research organization was established at that time. People 
who had extensive research experience were hired only as the need for research or evaluation 
expertise appeared. This staffing and organizational strategy made it difficult to formulate 
clearly Project goals and an overall research plan. 

In spring, 1973, three events changed the STD's strategy: 

1. The federal sponsor, the National Institute of Education, met with Project offi- 
cials and negotiated a clear set of Project goals. 

2. Project funding was reduced substantially. 

3. A newly-appointed Project director reorganized the STD and created a Research 
Component to handle all research and evaluation efforts. 

The Research Com^ionent was charged with designing, implementing, and documenting a comprehen- 
sive research plan that would address the new Project goals and also provide the information 
needed by sponsors and participants. 

DEVELOPING A FOUNDATION FOR RESEARCH AND EVALUATION 

The Research Component was supported fully by Project management, but it needed more than 
support to accomplish its tasks. It needed additional time and money. These resources were 
not available, because, as mentioned before, the Research Component was established after 
Project funds had been cut. It also needed a large and conpetent staff to handle the complex 
activities at hand. The diversity and quality of this input— compared to the available fiscal 
resources— dictated the creation of a research advisory board and the acquisition of continu- 
ing consultants. 

The advisory board reflected various perspectives and areas of expertise, including: re- 
search design and analysis; social action research methodology; educational measurement 



methodology; and coninunlcations technology. Board members were prominent figures in illa- 
tion: Dr. Dale Hamerus, director of the International Institute for Educational Technology, 
American International University, San Djego, California; Ms. Naomi Mcintosh, director of 
survey research, British Open University, London, England; Dr. Harold Mendelsohn, chairman 
of the Department of Mass Communications, University of Denver, Denver, Colorado; and Dr. 
Casper Paulson, Jr, director of programs for evaluation research. Teaching Research, Oregon 
State System of Higher Education, Monmouth, Oregon. 

From its inception, the board was given up-to-date information on Project goals and re- 
search activities. Its ongoing role was to provide advice on the: 

1. Determination of research policies and documentation strategies. 

2. Formation and implementation of a research plan. 

3. Resolution of evaluation problems. 

4. Interface with external evaluators. 

5. Identification of research consultants and related resources, if necessary. 

Board activities began in sunmer, 1974, when board members met with Project management and 
research personnel in Denver to help create a research and evaluation plan. The meeting and 
attendant activities outlined—in detail--Project goals, resources, constraints, and audiences. 

The Project goals, established June 8, 1973, in a "Memorandum of Understanding" to the 
National Institute of Education (the STD's federal sponsor) were: 

1. To demonstrate the feasibility of a satellite-based distribution system for 
rural, isolated populations. 

2. To test and evaluate user acceptance and the cost of various delivery modes 
using different materials. 

From a research perspective, these goals were broad and vague. Terms, such as "demonstrate," 
"feasibility," "distribution systems," and "acceptance," had not been defined. The goals 
for research purposes were redefined as follows: 

1. To collect, analyze, and interpret data which described and/or illustrated 
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the possibility or capability of developing, Implementing, and maintaining the 
Demonstration's hardware and support services. 

2. To collect, analyze, and Interpret data concerning the Internal and overt acts 
which Indicated agreement or approval by Individuals, groups, agencies, or In- 
stitutions of the Demonstration's hardware, support services, and courseware. 

3. To collect, analyze, and Interpret data concerning the costs of the hardware 
and support services which were relevant to the delivery of programs. 

Subsequent Input from participating states and sites led to the Inclusion of an additional goal 

4. To col test, analyze, and Interpret data concerning the educational benefits 
which were obtained by students. who participated In the Demonstration and 
who had different characteristics (age, grade, sex, achievement, and eth- 
nic).- 

Descriptive and quasi -experimental methods were used to obtain the Information detailed In 
the Project goals. Descriptive methods are extensive; they create a broad data base. These 
methods not only facilitate post-Project comparisons to Investigate specific cause-effect re- 
lationships, but they also provide baseline data for future planners who have the resources 
to conduct In-depth Inquiries and analysis. 

The decision to use descriptive rather than experimental methods to collect the data was 
based on these nonmutually-excluslve facts: ^ 

1. Ethnic and political constraints precluded random selection of populations and 
random assignment to treatment. 

2. The highly variable study conditions diminished the ability to control all 
treatments. 

3. Complex data analysis techniques were needed to analyze the research variables; 
these techniques required more sites per "cell" than were available. 

4. The exploratory nature of the Project precluded the anticipation of all Impor- 
tant research variables. 



-4- 



6 



SPECIFYING RESEARCH AND EVALUATION METHODS 

The STD decided to emphasize summative (descriptive) research for two reasons. First, 
soflie new products and services had been planned and developed prior to the implementation of 
the Research Component. Second, many new products and services had to be evaluated and tested 
in a short period. The STD further decided to meet the Project's formative needs on a prag- 
matic basis; to develop experimental methods only as the need for these methods appeared. 

A set of seven summative research studies Initially was designed to meet the Project's re- 
defined goals. These studies addressed the following: 

1. Did the technical delivery system work? 

2. Who initially used the delivery system? 

3. Who continued or discontinued to use the delivery system? 

4. Did the people who came and participated in the programs like the system 
and its products? 

1. Did the people who used the delivery system receive the intended benefits? 

6. What steps were taken by Project management to overcome the problems asso- 
ciated with implementing the delivery system and Its products? 

7. What costs were associated with the delivery system and its products? 

The initial research plan was submitted to the National Institute of Education (NIE)with 
the acknowledgement that the plan, though comprehensive, was regarded by Project personnel 
and advisory board members as the basis for further research and evaluation. The NIE sub- 
sequently approved the plan. This approval helped the STD to allocate staff resources to 
the studies and to assign research personnel to particular studies. During the next six 
months, all aspects of the summative studies were reviewed and refined. 

An essential aspect of the refinement period was the involvement of permanent, part- 
time consultants to provide Input and reaction to specific studies. Before obtaining a con- 
sultant, the Research Component developed a "work plan" which defined the needed Input; then. 
Research personnel asked the advisory board members to reconmend consultants who co'ild pro- 
vide the specified input. These recommendations helped the STD to select a group of 
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continuing consultants: Dr. Wilmar Bern thai , School of Business and Administration, Universi- 
ty of Colorado; Dr. Ron Corwin, Department of Sociology, Ohio State University; Dr. Phil 
Doughty, Department of Mass Communications, Syracuse University; and Dr. Thomas Drabek, Depart- 
ment of Sociology, University of Denver. Because the Project and research efforts were com- 
plex, the STD decided not to employ "one-shot" consultants. 

By February, 1974, the STD had consolidated the sumnative aspects of the research plan 
into six major studies. These studies centered on the following: 

1. Hardware and Hardware Support . This study addressed the broadcast and engin- 
eering elements of the satellite communications system and sought answers to 
these questions: 

(a) What is the quality of audio and video signals that participating sites 
receive? 

(b) What are the factors (weather and/or atmospheric conditions, satellite 
instability, and geographic location) whirh affect signal quality? To 
what degree do these factors Influence qualMy? 

(c) How reliable is the equipment that constitutes the delivery system? 
When repairs are needed, how responsive is the repair service? 

(d) Can site personnel with varied backgrounds be trained to use the com- 
munications hardware? 

(e) Are there factors, such as free-time activities and past work experi- 
ences, which relate to the relative effectiveness of site personnel 
in using the communica'tions hardware? 

2, Acceptance Study . This stucly addressed acceptance of the STD and its hard- 
ware, support services, and courseware by specific audiences. Major ques- 
tions for the study Included: 

(a) What was the student's level of acceptance for the STD and Its pro- 
grams, supplementary materials, and two-way voice communications? 

(b) What are the expressed opinions of audience members (teachers, admin- 
istrators, and conmunity members) at different sites to participating 
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in the Demonstration, viewing the programs, receiving instruction from 
the media, and using the two-way voice communications? 

(c) What do behavioral indicators, such as attendance patterns and resource 
allocations, suggest about a selected audience's acceptance of the STD? 

(d) Are there aspects of audience acceptance which can be explained by the 
type of site (Intensive, Receive-Oaly , and Open), by partici- 
pant demographics, and/or by the influence of others? 

(e) What are the levels of acceptance for the STD and its products and ser- 
vices among relevant decision-makers at state and local levels who are 
not direct audience participants? 

3. Student Benefits Study . This study addressed the career-related knowledge and 
attitude acquisition of participating junior high school students. The stu<ly. 
sought answers to these questions: 

(a) What was the average change/gain in participating students regarding— 

(1) career-related knowledge? 

(2) career-related attitudes? 

(b) How did the average change/gain benefits of participating students com- 
pare to change/gain benefits of nonparticipating students? 

(c) Are there differences in the students— 

(1) career-related knowledge? 

(2) career- related attitudes? 

(d) Can these differences be explained by— 

(1) type of site (Intensive, Receive-Only, or Open)? 

(2) student characteristics (age, grade, sex, achievement, or ethnic)? 

(3) affluence of school? 

(4) availability of support materials? 

(5) teacher acceptance? 

4. Case Studies ^ The purpose of this study was to observe carefully the "Dissem- 
ination-Adoption" process at three selected sites. Observations were guided 
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by these questions: 

(a) How were the sites selected? 

(b) How were the site preparations initiated? 

(c) What were the STD's positive/negative Inputs in terms of curriculums, re- 
source allocations, school procedures, and school/community relations? 

(d) What were the acceptance patterns among audience members to various STD 
products and services? 

(e) Did participation in the STD enhance/detract from the site's willingness 
to be involved in possible future efforts? 

5. Significant Events Stucty . This stucly addressed the significant Project events, 
issues, and decisions that were associated With planning, developing, implement- 
ing, and evaluating the Demonstration. Major questions for this stu((y Included: 

(a) What were the objectives of the Demonstration? How did thes* objectives 
evol ve? 

(b) What was the organizational structure of the Demonstration? How did this 
structure evolve? 

(c) What were the major managerial issues faced by the Demonstration? What 
strategies and procedures were devised to deal with these issues? Which 
strategies and procedures appeared most effective? What might future 
projects learn from the STD's experiences with a satellite-based com- 
munications system? 

(d) What were the major responsibilities of each STD component (Research, 
Broadcasting & Engineering, Prograrming, and Utilization)? How did 
each component organize itself to deal with these responsibilities? 
What problems (both anticipated and unanticipated) did they encounter? 
What might future projects learn from each component's experiences? 

(e) What was the nature of the relationships between the STD and its fed- 
eral-sponsoring agencies? What were the major issues In interacting 
with these agencies? What insights can be derived for structuring 
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future relationships? 
(f) -flow did the Demonstration address specific concepts, such as "Interaction," 
"user-based conmuni cations systems," and "regional approaches to common 
problems"? What insights can be gained from the STD's experiences In 
these areas? 

6. Cost Study . This study addressed the documentation and analysis of costs In- 
curred In selecting products and services. Emphasis was placed on developing 
and maintaining a comprehensive chart of accounts which would facilitate on a 
post-hoc basis: 

(a) A functional analysis with various aggregations of costs. 

(b) The projection of costs Into an operational system by amortizing costs 
across populations, time, products, and sites. 

By spring, 1974, the research effort centered on two areas. The first area focused on 
the identification and/or development and refinement of appropriate data collection Instru- 
ments and procedures for the summatlve studies. Ihe second was directed toward providing 
formative evaluation support for the development of educational television programming, par- 
ticularly the career development series for junior high school students. (See TR0210, The_ 
Formative Process Used by the Satellite Technology Demons tration in the Development of Tele- 
vision Programming for Junior High School Students for a a step-by-step description of the 
formative process.) 

COLLECTING AND ANALYZING DATA 

Data was collected (as needed) from STD sites and the region throughout the planning and 
development phases of the Demonstration. Data collection efforts* did not become systematic 
and Intense, however, until the STD began transmitting programs via satellite. These efforts 
were coordinated by the Utilization Component (or field service arm), which collected infor- 
mation about STD products and services in the field from state and site coordinators. Site 
personnel served at the local level to obtain data from school and community participants. 



-9- 

11 



as specified by the STD's research Instruments and procedures. 

Prior to the operational period, state and site personnel were provided training in all 
phases of their work; special attention was directed toward data collection and data integri- 
ty. Upon completion of training, site personnel helped to collect baseline data on: (1) 
participant awareness of the Demonstration; (2) participant expectations; (3) participant 
demographics; (4) student career-related knowledge and attitudes; and (5) school climates 
and organization. 

With the beginning of program transmissions, routine procedures were established for 
collecting data. Site coordinators sent weekly reports to the STD's headquarters in Denver. 
These reports consisted of: (1) daily signal quality measures and reports of any hardware 
malfunction; (2) daily or weekly student and teacher acceptance measures of STD products 
and services; (3) weekly audience acceptance measures of other products and services; and (4) 
a comprehensive weekly site report, addressing product effectiveness, product use, audience 
attendance, and STD support time. The weekly data flow provided Project management with a 
continuing source of information from the field. 

Additional data collection efforts were completed in accordance with the specifications 
in the research studies. These efforts included: (1) case-study observations; (2) a time 
series analysis of school staff acceptance; (3) a time series analysis of site operator 
hardware preferences; (4) midyear post-testing of first semester students and pre-testing 
of incoming second semester students; and (5) final post-testing of second semester students. 

In obtaining its massive data base, the research staff used multiple-pupose instruments 
and procedures, which covered the information needs of all the studies. This procedure meant 
that audience and respondent populations would not have to fill out duplicate fonns. 

Data handling and analysis procedures for the Demonstration included: (1) optical mark 
read (OMR) instrumentation, if possible; (2) a minicomputer for data file maintenance and 
descriptive level output; and (3) large computers external to the Project for major statis- 
tical outputs. 
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DOCUMENTING AND DISSEMINATING THE DATA 



With Input from the research advisory board, Project management and research adopted a 
documentation strategy that was designed to send information to many constituents who had ex- 
pressed interest in the Demonstration, The primary vehicles for this information flow were: 
(1) a brief, factual summary designed for mass dissemination; (2) an "executive" summary de- 
signed for moderate dissemination; (3) a series of 26 technical reports addressing specific 
topics and designed for distribution to selected audiences; and (4) a comprehensive final 
report designed for limited distribution. 

RESULTS 

The research effort conducted by the STD produced a major data base, which included: 

1. Quality of the ATS-6 signals and the factors that affected quality. 

2. Reliability of hardware. 

3. Performance and acceptance of hardware by site operators. 

4. Acceptance of received STD products by users anu the factors that 
affected acceptance. 

5. Acceptance of the STD and its products and services by other Project 
participants and relevant community and state decision-makers. 

6. Changes in career-related knowledge and attitudes by participating 
students and the factors that affected observed changes. 

7. Analysis of the events, procedures, personnel, and acceptance pat- 
terns that were associated with the participation of three selected 
STD sites. 

8. Analysis of the costs that were associated with the various phases 
of the STD and its components, products, and services. 

9. History of the Demonstration and the factors that affected develop- 
ment. 
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The steps taken to create the data base were considered as "interim" products. These 
steps included three comprehensive research designs, three formative evaluation models, and 
40-plus different instrument packages. 

None of these steps were accomplished without overcoming problems: Chief among these 
was the late involvement in the Project of a centralized Research Component. A second major 
problem was the incongruity between the scope of the effort and the available resources. 
The STD invested approximately 10 percent of its adjusted budget to formal research and eval- 
uation activities, excluding: (1) the contributions of the field services network (state 
and site personnel) to facilitate data collection; (2) the major roles of technical person- 
nel in hardware-related investigations; and (3) the contributions of Project management. 
This situation was made more acute by a failure to estimate adequately the time and re- 
sources that would be needed to address the Project's formative activities. 

The STD experienced not only internal, but also external problems. Conmuni cation with 
the National Institute of Education (the STD's federal sponsor) proved inconsistent and 
slow on matters relating to the initial research plan; the need for instrument clearances; 
the alleged opportunity to have early bids for external summative evaluations; and the need 
to obtain a final report from the external evaluations. 

SUMMARY 

The STD appears to have met the goals delineated in the "Memorandum of Understanding" 
with the'National Institute of Education. It designed. Implemented, and evaluated the STD's 
satellite-based communica^.ions system that was used to deliver a variety of products to 
a variety of users. Technical performance of the system was excellent, and most of the 
disseminated software products were rated good to excellent by users. Similarly, students 
who participated in the Demonstration showed significant gains over nonparticipants in 
desired subject areas. 

From a research standpoint, five of the summative studies were conducted as planned, 
and the resulting information base was equal to, or In excess of, the anticipated base. 
The sunnatlve efforts included were the hardware study, acceptance study, student benefits 
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study, case studies, and cost stu(ly. 

Clearly, the formative efforts exceeded that which was originally anticipated. In one 
of the summative studies, however, the information yields were less than planned. The area 
which was to address the creation of an interpretive history of the Project (the significant 
events study) was seriously weakened by the midyear departure, for personal reasons, of the 
principal investigator. 

RECOMMENDATIONS 

Based on its experience, the STD makes the following observations and reconmendations 
regarding similar research efforts. 

External 

1 . Federal Sponsors (the National Institute of Educationj .- 



(a) The posture of federal sponsors should be modified to reflect a more co- 
operative and involved stance toward project grantees. Although the adop- 
tion of such a posture has the inherent danger of nepotism, the NIE's 
current posture creates an adversary climate with inherent conmunlca- 
tions problems and inherent distrust. 

(b) Consistent with the first reconnendatldn, provisions should be made for 
more frequent (at least twice yearly) and less formal project visists 
by federal sponsors. 

(c) The NIE should be a source of counsel and advice on research matters. 
In the STD' s project, however, input was obtained only as a result of 
annual site reviews and could be described best as a "thumbs up or 
thumbs down" feedback. In the future, the NIE should establish a 
service arm to provide some technical support and liaison to the 
research efforts that It sponsors. 

(d) In the Interest of promoting the best possible projects and research 
efforts, the NIE should facilitate design, instrumentation, and pro- 
cedure exchanges among related projects. 
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2. External Evaluations , Having experienced two external evaluations— one fonnative 
and the other summative— the STD believes that its federal sponsors would have 
obtained more benefit by investing the additional resources in the original re- 
search. This observation not withstanding, the STD makes the following reco- 
mmendations regarding external evaluations: 

(a) If funds are provided for an external evaluation, they should be restricted 
to this purpose and focus on whether the project accomplishes agreed-upon 
tasks in a manner which supports the findings. 

(b) An ongoing project should have an opportunity to make recomnendatlons re- 
garding the contents of any request for quotations for external evalua- 
tions. In discussions about external evaluatioa contracts, bidders re- 
peatedly assorted that the request was misleading and did not take into, 
account Project research activities. The attempt by the NIE to maintain 
"separateness" generated much frustration, confusion, and initial dupli- 
cation of effort. 

(c) Products of external evaluations systematically should be available to 
project personnel; further, the manner in which this distribution is 
to be accomplished should be documented carefully in all operational 
plans. (The Educational Policy Research Center at Syracuse University, 
the STD's external evaluator, consistently shared status reports with 
the Project.) 

(d) Project personnel should be given an opportunity to provide written 
addenda to reports generated by external evaluators in order to 
support perceptions presented or to clarify positions on differing 
perceptions. 

(e) Budget negotiations should be given careful consideration at the in- 
itiation of the project to determine the type and extent of external 
evaluation that Is needed. 

(f) External evaluations should be reviewed on the basis of cost benefits. 
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Projects should be permitted to subcontract for an external verifica- 
tion of those elements that are deemed necessary by project sponsors. 

Internal 

1. Design . Research input should play a major role in planning a comprehensive 
project. Future decision-makers should insure that this input is acquired 
early in the project by hiring experienced research personnel or by creating 
an ongoing research advisory board. 

2. Staffing . In a comprehensive, but short-term project, it is difficult to obtain 
a staff with sufficient expertise to handle all the research tasks. Thus, pro- 
visions should be made, as in the STD, to support the internal research staff 
with selected ongoing consultants. Each staff member in the STD's Research 
Component, for example, was assigned to specific studies and tasks. As a 
result, unanticipated events and requirements (which emerged continuously) 

were dealt with as "overloads." Any large-scale project should possess at 
least two staff members beyond those needed for its summative studies: one 
to serve as project historian across all components; the other to facilitate 
data acquisition and data return to participants. 

3. Implementation . Many problems experienced by the STD's Research Component were 
the direct result of its relatively late implementation as a component. Inte- 
grating research personnel across aTl STD components did not provide the cohe- 
sion that was necessary to establish an overall Project research plan. It is 
recommended, therefore, that early implementation be incorporated into the re- 
search planning and development function. And, in projects that require a 
massive information base, it is further reconmended that this function be tied 
closely to management's activities. 

Relation to STD Users 

1. Data Flow . In the STD, data return from the sites could be accomplished through 
two channels: (1) directly to the Denver regional office; or (2) through the 
state coordinators to the regional office. Operational experience strongly 
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suggests that direct mailing Is the most expeditious channel, but not the most 
effective channel ror data flow; it does not facilitate as canplete data returns 
as does routing data through the state coordinators. It is rcconmended, there- 
fore, that in systems which require quick "turnaround," direct mall techniques 
should be used. But In systems which require canprehenslve data (for example, 
pre- and post-test data). Intermediary techniques should be used; all data 
should be routed through the official who has direct responsibility for site 
performance. 

2. Data Dissemination . Provisions should be Incorporated Into the operational and 
organizational plans to permit research personnel to disseminate data (at regu- 
larly scheduled Intervals) to participant groups. 

The above recommendations grew out of the STD's effort to plan and design research for 
a satellite-based comnuni cations system. Hopefully, the end of the STD's effort will mark 
the beginning of similar efforts in the future. The need for empirical data Is great. 



This report was produced with funding from the National Institute 
of Education » The views expressed do not necessarily reflect those 
of the National Institute of Education or the U.S. Department of 
Health, Education, and Welfare* 
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